Efficient CUDA

Recent work in deep learning has created a voracious demand for more compute cycles, but with the slowdown of Moore’s law, CPUs cannot keep up with the demand. Thus, attention has turned to massively-parallel hardware accelerators, ranging from new processing units designed specifically for deep learning to the repurposing of existing technologies like Fieldprogrammable gate arrays (FPGAs) and graphics processing units (GPUs). While these accelerators can provide immense speedups over traditional CPUs in certain domains, application programmers must take great care in optimizing code. In this work, we focus on GPUs and explore how 1) memory access patterns, and 2) memory abstractions can affect the performance of massively-parallel hardware accelerators, in the context of matrix multiply. We show that by optimizing memory access patterns, we can achieve over 15× speedups and choosing the wrong memory abstractions can lead to a 40× slowdown. Thus, we show the importance of memory in programming modern GPUs.